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By John Gaol and Shane Traflon 


The PSI Score Card: Determining 
When to Close a CTE Program 


TRADITIONAL ASSESSMENTS 
ARE INEFFICIENT IN HELPING 
PROGRAMS LEARN 
HOW TO LEVERAGE 
THEIR STRENGTHS AND 
COMPENSATE FOR THEIR 
WEAKNESSES. 
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| | OW DO YOU KNOW 

U WHEN IT IS TIME TO El- 
I I mm GIVI A PROGRAM 
EXTRA ATTENTION OR PERHAPS 
CLOSE IT? Thai problem faces many 
career and technical education (CTE) 
programs and there is. not necessarily 
a oxic-4i3tfr-fiU-aU solution. Traditional 
a xse s s.n i cm s are in cil ic ie n c i n help i i lg 
programs learn how to leverage their 
siren gibs and compensate far [heir weak- 
ness es. In Si. I XHiis C o u nt y , iV I is so it r i , a 
i ca a 1 of CT E advise rs iu id a d m i n ist i 1 aiors 
arrived at a model for reviewing pro- 
grams f&c ing this dial I en gi 

The St. Louis County Spec: in I School 
DbLricc (55 Dj ban overlay db irk i that 
provides Imih special education and CTE 
in its 23 school disl ricis. (i l'K services 
arc provided through SSITs two techni- 
cal high veil cols: North Technical H igh 
i I s 20 0 s i u de r Us) a n d S o nth' lech n i c a I 
J ligli [5.10 student &). The district offers 34 
rarrn and technical programs with 
being duplicated at both schools. 

SS D 5 s 5 l* p e i 1 in leu dr n i , j oh n 0 a r y 3 
and bis CTE advisory board wanted Ic 
use data in evaluate the district's CH E 
programs. A subcommittee was farmed to 
look Ltiln pitjgrani initial inn, evaluation 
a aid tea 1 m m m s< n i, ' V h v sup C r i n I nude n| a rid 
I he hoard members, were locking Fora 
lorn ltd progrsa.ni that consisted of a writ- 
ten policy, and subsequent procedures, 
i s ) e v a lu ate ah of die technical ed u cation 
programs. 

Doing the Groundwork 

hirst, the subcommittee investigated 

whether other career centers and/or 

career districts had a forma] system of 


evaluation for carver and technical pic:- 
grains. After contacting ^ I issuin' i career 
cciucrc find finding no formal writ urn 
processes, calls wem out to other slates." 
CTE centers. Sandra Royer, the health 
and. consumer sciences supervisor at 
M jam i Va Hey C a rc c r T ee hi ioI ogy C W nt er 
in Clayton, Ohio, provided a draft called 
Disinvestment Guidelines. The guide- 
lines called (or programs ihal fell below 
el certain level of enrollment to be placed 
on a watch list. Programs that continue so 
fa 1 1 on t Ji c list, a fie r at la I y s b a i a d i i it er- 
veiuiQHS, could be placed on probation, 

T1 k mgl i t h ere were i it j fo r m a l gu idc 1 i i ic s- s 
Rovei - indicated ihai it was a good place 
to start a conversation to evaluate their 
programs. 

js The primary purpose of any system- 
atic assessment of school performance... 
is Ic reveal best prac I ices mid identify 
shared problems to order to encourage 
teachers and sekonh lo develop more sup- 
port ivc and productive learning pnrirmi- 
niciiLs." surest Schleicher and Stewart 
[20 [If!, |j 49). Accordingly, l he subcom- 
mittee liked the idea of a tiered system 
in evaluating I he programs. A program 
would hr placed on a watch list and if 
things didn't improve the second year, il 
would hr placed on probation- In future 
years, the program could be term mated 
if progress, is not attained, A duplicated 
program, mch as carpentry, would tic 
c val u.a C ed u £.111 g s cp ei j "a LC dM n (br e ;- 1 c 1 1 
school. 

Using the PSI Scorecard 

The ttctfi step was lo find valid and mea- 
surable criteria to (rv&l pale the programs, 
krervtz {2008, p. 19) posits, : A sensible 
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"THE PRIMARY PURPOSE OF ANY SYSTEMATIC 
ASSESSMENT OF SCHOOL PERFORMANCE JS TO 
REVEAL BEST PRACTICES AND IDENTIFY SHARED 
PROBLEMS IN ORDER TO ENCOURAGE TEACHERS 
AND SCHOOLS TO DEVELOP MORE SUPPORTIVE 
AND PRODUCTIVE LEARNING ENVIRONMENTS/' 
SUGGEST SCHLEICHER AND STEWART. 


accountability system, , .holds people 
accountable for what they can control” 

To this end. the subrommiticc members 
s La r lot I by linting all rsf il te ike tor ft t he y , as 
stakeholders, tliivughi should Ik l assessed 
in order to operate a .successful t i I K pro- 
gram, After much discussion, the subcom- 
mittee combined many of die factors and 
came to concensus on measurable criteria 
which relate to federal and state account- 
ability standards. 

The criteria used i <1 evaluate the 
programs arc called Program Status hi- 
d sea tors I l ). 1 1 k sub com mil ie c sell I ed 
cm five major ESI which provide sufficient 
objective data to analyze a progr^mV 
success. They arc; phrcemciii, enrol Imcni, 
advisory comm iuecs 3 cerlrfica.lLon, and 
occupational outlook. What follows are 
the delinkinus of the ESI. as well asciimlT 
scores, identified lor each indicator 

PI a ee merit- A r n ml>i nat inn of jo] > and 

j >[> s is cc o n d a r y p I a oem e n l is re vie wed . 
Positive Missouri School Improvement 
L Vr igra m A I Si P) p lac e i tip nisi n-cl nd e 
students: 

L working in their field of study 
or a related field; 

2. continuing past secondary itudy; m L 

3. entering the military. The data is 
taken from the previous year’s 1 8 flu 
day follow-up surveys. 

Advisory Crtrrixriit terx; Following 

the newly approved Missouri Workforce 
I n vest m e i u B oai ’d V gu i rie I i nes pc n lining 


to i lie makeup of the body, 51 percent 
of advi sory members must l>e business/ 
industry mpraaenuuiveH. Business rep- 
resentatives arc those in a business with 
Kiip cr v S sory/d cc isi on -n i akii lg role s . For 
the 2009-2010 school year there must be 
at least eight advisers, with live advisers 
present ill an admory meeting; in the 
2< ) 1 0-2 Oil s ( ;hi >i A yea r t li c rr must b v I [] 
advisers, wiLh sis pres ecu at eiii advisory 
meeting; in (he 2(11 1-2012 school year 
i I k re m i ist I k 1 2 nd v isc rs, with v cn 
jtu'iuIkts present. In order to have a 
quorum, a majority of the present advisers 
in u h 1 > c b u s i n ess/i n di tst ry rc| Trcscn t it t i ve. $ . 

Enrollment Fifty percent of Student en- 
rollment cap si city needs t ci hr maintained. 

' i ' h c e c* Lai c ap act c y o f a progi a mine lud cs 
1 1 o 1 1 1 ju it u n m a nd s cm ors. 1 1 ' cap a c it y for 
the junior class is 20 st uric iris and t hr 
capacity lor the senior class is 20 stride ills, 
llie total capacity is 49 students, Programs 
! ice d 50 per ce nt e n ml 1 1 nr n t to qua ] i I y for 
\ l isis a u r i 1 ^ q >a it mem o f Elc m en tar y a n d 
S it on dn r v E due a t m n i re im burs cm r ti i 

Certification: All programs will be 
required tn have 100 percent nf their 
student^ take a Technical Skill Assessment 
i ' J \S A ) I ly the spring of 2012 in nrd er I n 
fiilfill the IV 1 rk ills Icdernl legislation. The 
U . S . Depart m m t of Ed ticatio n has stt 
the pass rate at 62.5 percent for second- 
ary GTE Students, The SSD Technical 
High School currently ad m in mm the 
W hrkKcys. test tu all incoming juniors and 
exiting sen ion. The Work Keys' Career 


Readiness Certificates will be used to 
in casu re s t u d i: in meets s for a.3 I p rograi 1 3 s 
that do nm have ;m cm bibbed TSA in 
place, 

Occupational Outlook: Either the 

U.S. Depart mem of Luboris Occupa- 
tion! Ouihmk Umif/boat i > r the Missouri 
Economic Research and Information 
Center MERIC, is consulted to clas- 
sify occupational projections. I ERIC 
was selected us- the primary source since 
career projections arc provided in a grade 
format: Grade tl A”=ExcclIenL Outlook 
u> l ‘F vt =Ponr Outlook, (There arc five 
grade a/ 1-evelO 

In addition to establishing a baseline (dr 
each indicator, the sulKommiU.ee wanted 
u> dcvchp a rubric where all qf the KS! 
could be weighted in order to come up 
with, a composite score. In this way, no 
single PSI would cause a program to 
lie term in a ted a i id a 1 1 five incite: at ays 
would he considered when cvahisilmc a 
program's staLus. AIU r much discussion 
a i id mi imi Imi s , ih c Ee il ] t >w i e ig ] n si ills and 
percentages were aligned to each ESI; 


Figure! 



30 percent 

30 points 

Enrolfment 

25 percent 

25 points 
20 points 

J^dviMiy Commirree 

20 percent 

Certification 

20 percent 

20 points 
5 points 

Otcupfitionfli Outlook .... 




As a sample exercise using t lii^ system, 
dnia from the 2007-2008 schoo] year was 
examined lo deter mine where programs 
would fill within ihe rubric 'Sec figure 
I). Within the data chart you arc able 
to sec how the ESI am scared for each 
program. If a program had a placement 
rate of 100 percent, it would earn ;*tll 30 
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I Klims, [fa program had a placement rate 
of 50 percent, it would earn 15 points 
in liiis category. The same principle of 
proportional scoring applies to the other 
categories. J fen roll mem is 50 permit of 
capacity, then the program would earn 13 
of the 25 points {rounded up). For the sake 
t}i 1 e %\ ah 1 1 sh i tig a tot pi r o m\ jm i I e r core 
cutoff lor the rubric, it is assumed dial all 
programs will have two advisory comm Si- 
te e mec lilies e i ich yea rw it h s h e a| >i>ro[ >r i - 
ate: number of advisers. I f for sc im e re as on 
a program had c silly one advisory meeting 
ihai met the cbtabLUlicd criteria. then the 
program would only receive lOolThc 2 13 
pot ms available. The subcommittee *cu 
die certification iiidieainr hi 70 percent 
: 34 points} only ina the sake of establish- 
ing a composite rubric score. The out look 
indicatory range from 5 to i based on the 
A r IT C, D. Y grade categories established 
hyMERKk 

After ah of I fir data from each pro- 
gr^iii was put into the sample rubric, 
there wen? 13 of the 51 programs evaluat- 
ed (about 20 percent} that w re below the 
composite cutoff score of 70 percent. The 
committee judged that this was a good 
place to start. The tutolTseore can always 
lie changed by the CTE advisory board 
depending upon changes in the PEL All 
programs that hill below the cutoff store 
of 70 will he on the official watch list. Any 
program cn flic list is required to write 
an improvement plan tailored to meet 


the needs of die identified problem areas. 
The plan will be written with, input from 
t cue he rs a a dmi n h\ trati >rs Eiiacl mlvi sory 
team members All plans will be submit- 
ted to the building administrators for 
approval iind lorwanthd to the superin- 
tendent's CTE advisory boat'd. 

Program a that are not on the watch 

list bite have one nr more l 3 Sl which fall 

hehm acecptiihle levels will be required to 
have a n imp rov cm c n t p la 1 1 m ibr nit .tec S i o 
the building administrators. The k-ssthan 
acceplnbie levels aro aft lt»! lows: 

Placement ........... bO percent 

Enrollment — . 50 percent 

Advisory 3. 1 ec i i i lgs ......... 1 00 pe item 

Certification 02 5 permit 

Program Termination is □ 
Three-tiered Process: Watch — 

P robot i o n — Ter mi n a ti on 

I -.ad i year die CT E ad v isor y bo a rd w ill 
examine the impTCni'iltiTiE plans uf the 
programs on the w^trh and probation 
lists. Programs, ilia! continue to per form 
bdOw acceptnhD level? may be reotim- 
i n c tided for termination by ihe CTE 
advhtin hoard in the district's board of 
education. SSL) will implement I lie PS1 
me del in the upcoming 2009-2010 school 

year. It is SSD^ intcjllitm U> report hack 

to CTE professionals in throe years and 
provide results of I his endeavor. Hopeful- 
ly. m tide nis ? cduea tors and businesses will 


be nefit from SS I.) ! s at i cmj u co u I Hi av t Lu a 
to improve outcomes based on, an array of 
mutual interests. 

The subcommittee member? were: 

John Gaal, director of training and work- 
force development for the Carpenters’ 
District Council of Greater St. Louis; Jeff 
S pi ege I, nli| jer intend en i of i be Ferg uson- 
Florimni School District in St. Louis 
County; George Les.li era, owner and 
president of [PC Graphics Priming and 
Design; Mike Powers, principal of North 

Technical Hi^li School; K.ini ford- heals, 
assistant principal of Non 1 1 Technical 
Hijrh School; Randy Barnes., Ilioliuitor of 
technology lor the SSP Technical Educa- 
tion Division; Jim Fischer, retired SSI) 
t echnie al e due a t i e n ad m i n i st r at n q a n d 
Shane Trafton, administrator of currieu- 
lu 1 1 1 and in sir ue l io e i lor the S S D Tec: 1 1 1 1 i~ 
cal Education Division. D 
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